Systems and methods for classifying electronic documents

ABSTRACT

A method of classifying an electronic document from an electronic media source includes generating statistical data from one or more training documents and creating a plurality of classification rules, including creating at least one topic model-based classification rule using the statistical data and creating at least one query-based classification rule using one or more user defined categories and the statistical data. The method further includes classifying the electronic document using the at least one topic model-based classification rule. Example systems for classifying an electronic document from an electronic media source are also disclosed.

FIELD

The present disclosure relates to systems and methods for classifyingelectronic documents.

BACKGROUND

This section provides background information related to the presentdisclosure which is not necessarily prior art.

Document classification typically includes assigning documents to acategory. The documents may include, for example, texts, images, music,etc. The documents may be classified according to the subject mattercontained therein by content-based classification and request-basedindexing. Content-based classification is classification in which theweight given to particular subjects in a document determines the classto which the document is assigned. Request-based indexing isclassification in which requests from users (e.g., particular topics,etc.) influences how documents are being classified. Typical approachesto classify documents may include, for example, Naive Bayesian, supportvector machines, decision tree(s), logistic regression, etc.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

According to one aspect of the present disclosure, a method ofclassifying an electronic document from an electronic media source isdisclosed. The method includes generating statistical data from one ormore training documents and creating a plurality of classificationrules, including creating at least one topic model-based classificationrule using the statistical data and creating at least one query-basedclassification rule using one or more user defined categories and thestatistical data. The method further includes classifying the electronicdocument using the at least one topic model-based classification rule.

According to another aspect of the present disclosure, a non-transitorycomputer readable medium storing computer-executable instructions forcreating a plurality of classification rules to classify an electronicdocument from an electronic media source is disclosed. The instructionsinclude generating statistical data from one or more training documentsand creating a plurality of classification rules, including creating atleast one topic model-based classification rule using the statisticaldata and creating at least one query-based classification rule using oneor more user defined categories and the statistical data.

According to still another aspect of the present disclosure, a computersystem for classifying an electronic document from an electronic mediasource is disclosed. The computer system includes a communicationnetwork and a computer server in communication with the communicationnetwork. The computer server is configured to receive the electronicdocument via the communication network. The computer server has memoryand a processor. The memory includes one or more training documents andone or more user defined categories. The processor is configured togenerate statistical data from the one or more training documents andcreate a plurality of classification rules. The plurality ofclassification rules include at least one topic model-basedclassification rule created using the statistical data and at least onequery-based classification rule created using the one or more userdefined categories and the statistical data. The processor is furtherconfigured to classify the electronic document using the at least onetopic model-based classification rule.

According to yet another aspect of the present disclosure, a method ofclassifying an electronic document from an electronic media source usingat least one classification rule from a plurality of classificationrules is disclosed. The classification rules include at least one topicmodel-based classification rule and at least one query-basedclassification rule. The topic model-based classification rule iscreated using statistical data generated from one or more trainingdocuments and the query-based classification rule is created using thestatistical data and one or more user defined categories. The methodincludes classifying the electronic document using the at least onetopic model-based classification rule.

Further aspects and areas of applicability will become apparent from thedescription provided herein. It should be understood that variousaspects of this disclosure may be implemented individually or incombination with one or more other aspects. It should also be understoodthat the description and specific examples herein are intended forpurposes of illustration only and are not intended to limit the scope ofthe present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations, and are notintended to limit the scope of the present disclosure.

FIG. 1 is a block diagram illustrating a method for classifying anelectronic document from an electronic media source according to oneexample embodiment of the present disclosure.

FIG. 2 is a flow diagram illustrating a method for creating one or moreclassification rules according to another example embodiment.

FIG. 3 is a flow diagram illustrating a method for classifying one ormore electronic documents using the classification rules created in themethod of FIG. 2.

FIG. 4 is an example topic-term matrix including topical categoriesemployed to create the classification rules in the method of FIG. 2 andclassify the electronic documents in the method of FIG. 3.

FIG. 5 a is an example graph illustrating probabilities of particulartopical categories of the topic-term matrix of FIG. 4 being found in anexample article.

FIG. 5 b is another example graph illustrating probabilities ofparticular topical categories of the topic-term matrix of FIG. 4 beingfound in another example article.

FIG. 6 is a screen shot illustrating example topic model-basedclassification rules in XML format created in the method of FIG. 2.

FIG. 7 is a screen shot illustrating example query-based classificationrules in XML format created in the method of FIG. 2.

FIG. 8 is a block diagram of a system including a processor forclassifying an electronic document from an electronic media sourceaccording to another example embodiment.

Corresponding reference numerals indicate corresponding parts orfeatures throughout the several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference tothe accompanying drawings.

Example embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. It will beapparent to those skilled in the art that specific details need not beemployed, that example embodiments may be embodied in many differentforms and that neither should be construed to limit the scope of thedisclosure. In some example embodiments, well-known processes,well-known device structures, and well-known technologies are notdescribed in detail.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting. As usedherein, the singular forms “a,” “an,” and “the” may be intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. The terms “comprises,” “comprising,” “including,” and“having,” are inclusive and therefore specify the presence of statedfeatures, integers, steps, operations, elements, and/or components, butdo not preclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. The method steps, processes, and operations described hereinare not to be construed as necessarily requiring their performance inthe particular order discussed or illustrated, unless specificallyidentified as an order of performance. It is also to be understood thatadditional or alternative steps may be employed.

When an element or layer is referred to as being “on,” “engaged to,”“connected to,” or “coupled to” another element or layer, it may bedirectly on, engaged, connected or coupled to the other element orlayer, or intervening elements or layers may be present. In contrast,when an element is referred to as being “directly on,” “directly engagedto,” “directly connected to,” or “directly coupled to” another elementor layer, there may be no intervening elements or layers present. Otherwords used to describe the relationship between elements should beinterpreted in a like fashion (e.g., “between” versus “directlybetween,” “adjacent” versus “directly adjacent,” etc.). As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items.

Although the terms first, second, third, etc. may be used herein todescribe various elements, components, regions, layers and/or sections,these elements, components, regions, layers and/or sections should notbe limited by these terms. These terms may be only used to distinguishone element, component, region, layer or section from another region,layer or section. Terms such as “first,” “second,” and other numericalterms when used herein do not imply a sequence or order unless clearlyindicated by the context. Thus, a first element, component, region,layer or section discussed below could be termed a second element,component, region, layer or section without departing from the teachingsof the example embodiments.

Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,”“lower,” “above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. Spatiallyrelative terms may be intended to encompass different orientations ofthe device in use or operation in addition to the orientation depictedin the figures. For example, if the device in the figures is turnedover, elements described as “below” or “beneath” other elements orfeatures would then be oriented “above” the other elements or features.Thus, the example term “below” can encompass both an orientation ofabove and below. The device may be otherwise oriented (rotated 90degrees or at other orientations) and the spatially relative descriptorsused herein interpreted accordingly.

A method of classifying an electronic document from an electronic mediasource according to one example embodiment of the present disclosure isillustrated in FIG. 1 and indicated generally by reference number 100.As shown in FIG. 1, the method 100 includes, in block 102, generatingstatistical data from one or more training documents and, in block 104,creating classification rules. The classification rules include at leastone topic model-based classification rule created using the statisticaldata and at least one query-based classification rule created using oneor more user defined categories and the statistical data. As shown inFIG. 1, the method 100 further includes, in block 106, classifying theelectronic document using the topic model-based classification rule.

By creating the classification rules using the same statistical datafrom the training document and the user defined categories (sometimesreferred to as “ad hoc categories” or “user defined ad hoc categories”),the classification rules become focused on topics relevant to users.Because the classification rules are focused on relevant topics,classification of the electronic document using the topic model-basedclassification rule becomes more relevant and, in turn, data from theclassified electronic document may be more effective and useful to theuser.

For example, users are able to identify topical categories discussed indocuments relevant to their industry, expertise, etc. and particulardocuments discussing their ad hoc categories related to their brand,product, etc. This allows users to identify topics and documentsrelevant to the users and, in turn, to utilize this information to moreaccurately collect data pertaining to their business, interests, etc. Asa result, users may be able to monitor relevant discussions (e.g.,positive and/or negative discussions regarding a product, falseaccusations, etc.), monitor new topics and/or issues, etc. via documents(e.g., blogs, messages, emails, etc.) representing the discussionsand/or respond to disgruntled customers, alter protocols, etc. based onthe discussions. This may be particularly important with the increasingpopularity of social media which gives consumers a forum to share theirexperiences, opinions, etc. regarding products and services with otherinterested parties.

The topic model-based classification rule may be created by astatistical model (e.g., topic model, etc.) and the query-basedclassification rule may be created by a search engine model (e.g.,Lucence Engine, etc.). The statistical model may generate thestatistical data from a training document. For example, the statisticaldata may represent topics from the training document. A topicmodel-based classification rule is then created using the statisticaldata.

A query-based classification rule may be created (at least in part) byusing user defined categories. For example, a user may define particularcategories relevant to the user and the search engine model may thencreate a query-based classification rule using the statistical data(e.g., generated by the statistical model) and the user definedcategories. Accordingly, the statistical model may provide relevant datato the search engine model to create the query-based classificationrule. In this way, the statistical model may supplement the searchengine model with statistical data to create the query-basedclassification rule. As a result, the classification rules becomefocused on topics more relevant to users.

Additionally, if both models utilize the same statistical data, the rulecreating process and the classification process may become moreefficient, allowing more electronic documents to be classified within aparticular time period.

Alternatively, and as further explained below, the method 100 mayinclude classifying the electronic document(s) using both the topicmodel-based classification rule and the query-based classification rule.By doing so, classification of the electronic document may be moreeffective and the resulting data from the classification may not onlyincrease but also may be more relevant to a particular user.

FIG. 2 illustrates an example method 200 of creating one or moreclassification rules. In the example of FIG. 2, the method 200 isexecuted offline (e.g., disconnected from a communication network).Alternatively, the method 200 may be executed online (e.g., connected toa communication network).

As shown in FIG. 2, the method 200 includes receiving training documents(block 202). In some embodiments, the training documents may includeunlabeled training documents that are not previously analyzed,classified, etc. into a certain category by individuals. Accordingly, ifunlabeled training documents are employed, previous knowledge about thedocuments may not be necessary and/or provided.

Although the example of FIG. 2 illustrates receiving multiple trainingdocuments, only one training document may be received without departingfrom the scope of the present disclosure. Preferably, multiple trainingdocuments are employed so that the classification rules are morereliable. Additionally, the training documents may be representative ofelectronic documents that will be classified based on the classificationrules (further described below). This may result in more effectiveclassification rules.

In the example of FIG. 2, the training documents are processed in block204. For example, each training document may be filtered to cleantextual content, including, for example, translating words to a desiredlanguage, stemming words to a root word, removing stop words (e.g., the,is, at, which, on, etc.) and/or low frequency words, etc. By filteringthe training documents, efficiency of the statistical model and thesearch engine model may be increased. Alternatively, the trainingdocuments may not be processed and instead the method 200 may skip block204 and utilize unprocessed training documents to create theclassification rules.

As shown in block 206, the method 200 employs a topic model engine asthe statistical model. The topic model engine receives the processedtraining documents (of block 204) and is able to discover topicalcategories discussed in the processed training documents. The number oftopical categories may be preset to a constant value for each trainingdocument. The topic model engine may generate statistical data (e.g.,using the topical categories) from the processed training documents. Inthe example of FIG. 2, the statistical data includes a document-topicmatrix (shown in block 208) and a topic-term matrix (shown in block210).

Because the topic model engine discovers the topical categoriesdiscussed in the training documents, the topical categories in thetraining documents do not need to be manually classified prior toinputting the training documents into the topic model engine.Accordingly, knowledge regarding potential topical categories found inthe training documents may not be required in advance. Thus, unlabeledtraining documents (as explained above) may be employed. Additionally,because no advance knowledge regarding potential topical categories maybe needed (besides the number of topical categories as explained above),the topic model engine can adapt to topical changes and discover newtopical categories as needed.

In the example of FIG. 2, the topic model engine may be any suitabletopic model engine including, for example, a probabilistic latentsemantic indexing (PLSI) model, a latent Dirichlet allocation (LDA)model, etc.

The topic-term matrix (shown in block 210) generated from the topicmodel engine is a probabilistic distribution of each topical categorywithin the processed training documents over the terms (e.g., words) inthese training documents. These probabilities put a weight on each termin the training document with regard to its relevance to the topicalcategories. For example, a term having a higher probability value ismore likely to be chosen by an author discussing a given topicalcategory than a term having a lower probability value.

FIG. 4 illustrates an example topic-term matrix 400 that may be createdby the topic model engine. In the example of FIG. 4, the topic-termmatrix contains fifteen topical categories in random order (shown asTopics 1-15). A name for each topical category may be manually addedbased on the terms provided in a particular topical category. The namesmay be general so that more than one topical category has the same name.For example, as shown in FIG. 4, the Topic 4 and the Topic 5 are bothnamed “Energy.” Alternatively, the names may be specific. For example,the Topic 4 includes terms related to traditional energy and thereforemay be named “Traditional Energy” and the Topic 5 includes terms relatedto green energy and therefore may be named “Green Energy.”

For each topical category, twenty terms are provided in a column.Alternatively, more or less topical categories and/or terms may beemployed. The terms are ordered according to the probability that aparticular term is used when discussing a particular topical category.For clarity, the probability value for each term is not shown. Forexample, the term “food” has a higher probability of being used whendiscussing Topic 3 than the term “organic.”

Referring back to FIG. 2, the document-topic matrix (block 208)generated from the topic model engine (block 206) may be a distributionof the topical categories from the topic-term matrix (e.g., shown inFIG. 4) for each training document. The prominence of a particulartopical category in a document may be represented with a probabilityvalue from zero (0) to one (1). A topical category having a higherprobability value means it has a higher relevance to a particulardocument.

FIGS. 5 a and 5 b are graphs 500 and 502, respectively, representingsample document-topic matrixes generated from the topic model engine.The graphs 500, 502 illustrate probabilities of particular topicalcategories of the topic-term matrix of FIG. 4 being found in exampledocuments. As shown in the example of FIG. 5 a, the graph 500illustrates a distribution of topical categories for the followingexample document (from, e.g., a media source):

-   -   Starbucks Via Ready Brew Coffee costs about 1/10th of the price        of a cup of coffee from a Starbucks store and contains anywhere        from zero calories to upwards of 100 calories, depending on the        flavor you get. The portable packets make it easy to enjoy a cup        of coffee anywhere! Simply pour the packet into a cup, add hot        water and stir! The Ready Brew line of coffee comes in 14        different flavors, from your regular Italian Roast to Ice        Caramel to French Roast! Save calories, time and money! Product        Features 100% Arabica coffee only 59 cents per packet from        Amazon.com! Just add hot water and stir. Available in 14        flavors. Where to buy: Ready Brew coffee is available at your        nearest Starbucks or you can have them shipped to you for FREE        from Amazon.com! Also check out: 3 Tips To Healthy & Delicious        Coffee It's no secret that I'm a big fan of coffee!        As shown in the example of FIG. 5 a, the above document includes        three topical categories (from the topic-term matrix 400 of        FIG. 4) having a relatively high probability. The topical        categories include Topic 15 (“Food”) having a probability value        of about 0.475, Topic 13 (“Health”) having a probability value        of about 0.25, and Topic 9 (“Online Shopping”) having a        probability value of about 0.21. Thus, the example document is        deemed relevant to the Topic “Online Shopping”, more relevant to        the Topic “Heath”, and even more relevant to the Topic “Food.”

The example graph 502 of FIG. 5 b illustrates a topical categorydistribution of the following example document:

-   -   Chrysler to sell natural gas-powered truck: Chrysler aims to be        the first U.S. automaker to produce a factory-built pickup truck        that is powered mainly by natural gas. The privately held        company said Tuesday that its new Ram 2500 Heavy Duty CNG truck        will be sold to commercial customers that operate truck fleets.        The company expects to deliver the first trucks in July. The        truck will have natural gas tanks and an 8-gallon fuel tank for        gasoline. Chrysler said a small amount of gasoline is needed to        start the truck, but after ignition it runs entirely on natural        gas. If the natural gas tanks run out, the engine can switch to        gasoline. Natural gas prices have dropped steeply over the last        year thanks to higher production. Chrysler said the gas-powered        trucks will save money over the long term.        As shown in the example of FIG. 5 b, the above document includes        two topical categories (from the topic-term matrix 400 of        FIG. 4) having a relatively high probability. The topical        categories include Topic 5 (“Energy”) having a probability value        of about 0.475 and Topic 12 (“Auto”) having a probability value        of about 0.4. Thus, this example document is deemed relevant to        the Topic “Auto” and more relevant to the Topic “Energy.”

Referring back to FIG. 2, the method 200 further includes (in block 212)creating one or more topic model-based classification rules based on thedocument-topic matrix (block 208) and the topic-term matrix (block 210).For example, the topic model-based classification rules may be createdby analyzing semantics of each topical category based on relevant terms(of each topical category) having the highest probability from thetopic-term matrix (block 210) and by analyzing a distribution of eachtopical category from the document-topic matrix (block 208). Particulartopical categories may then be identified and placed into one or morecategories of interest. A category of interest may include particulartopical categories that may be relevant to a particular user. Thecategory of interest may be broad (e.g., entertainment, sports, etc.) ormore specific (e.g. movie, baseball, etc.). For example, if a user isinterested in the entertainment industry, a category of interest mayinclude Topic 6 (“Entertainment”) and Topic 10 (“Entertainment”) of thetopic-term matrix of FIG. 4.

After identifying the topical categories relevant to a category ofinterest, one or more topic model-based classification rules may becreated to assign an electronic document to the particular categories ofinterest. For example, a topic model-based classification rule may be“IF the probability value of Topic 6 OR Topic 10 is above a certainthreshold for document A, THEN document A is assigned to theEntertainment category of interest.”

The topic model-based classification rules created in block 212 may thenbe applied to and evaluated on one or more test documents. This is shownin blocks 214 and 216. The test documents may be randomly selected fromthe training documents described above. Based on the topic model-basedclassification rules created in block 212, each test document may beclassified (e.g., assigned to a category as further explained below).This is shown in block 214.

The classification results from block 214 may be evaluated in block 216to ensure the topic model-based classification rules are suitable. Insome embodiments, an individual may analyze the text of the testdocuments and ensure each test document is correctly classified. Thetopic model-based classification rules may then be evaluated byutilizing, for example, precision and/or recall equations to ensure thetopic model-based classification rules meet a defined threshold.Equations (1) and (2) below are example equations for determiningprecision and recall, respectively.

$\begin{matrix}{{Precision} = \frac{N\left( {d_{c} +} \right)}{{N\left( {d_{c} +} \right)} + {N\left( {d_{c} -} \right)}}} & (1) \\{{Recall} = \frac{N\left( {d_{c} +} \right)}{N\left( d_{c} \right)}} & (2)\end{matrix}$

In the example equations (1) and (2), N(d_(c)+) is the number of testdocuments correctly classified to category C based on the topicmodel-based classification rules, N(d_(c)−) is the number of testdocuments incorrectly classified to category C based on the topicmodel-based classification rules, and N(d_(c)) denotes the total numberof test documents that should be classified to category C based on humanjudgment.

If the results of the precision equation and/or the recall equation arebelow the defined threshold, the topic model-based classification rulesmay be deemed unsuitable. As a result, the topic model-basedclassification rules may be modified. The modified classification rulesmay be applied to the one or more test documents and evaluated again asexplained above. This process of evaluating the topic model-basedclassification rules may be repeated until the topic model-basedclassification rules meet the defined threshold.

The topic model-based classification rules determined to be suitable maybe identified and stored for future use as further explained below. Forexample, the topic model-based classification rules may be formatted andsaved in XML files.

FIG. 6 illustrates a screen shot 600 of example topic model-basedclassification rules in XML format. As shown in FIG. 6, the screen shot600 includes three topical categories (Energy, Auto and Entertainment).A description of each topical category, a list of matched topics and aprobability threshold for each matched topic are shown below eachtopical category. For example, the Energy category includes adescription of “Topics about both traditional and green energy” and hastwo matched topics. The matched topics include Topic 4 and Topic 5 (ofFIG. 4) that refer to green energy and traditional energy, respectively.The probability threshold for Topic 4 and Topic 5 is 0.20 and 0.30,respectively. Thus, if a document has a probability value above 0.20 forTopic 4 or above 0.30 for Topic 5, the document is classified under theEnergy category. Therefore, based on this topic model-basedclassification rule, the example document of FIG. 5 b would beclassified into the Energy category because the document includes aprobability value of about 0.475 for Topic 5 (as explained above).

Referring back to FIG. 2, the method 200 further includes creating oneor more query-based classification rules. In the example of FIG. 2, themethod 200 employs a Lucene Engine (created by Apache Lucene™) as thesearch engine model to create the query-based classification rules. Thisis shown in block 226. Therefore, each query-based classification ruleis a Boolean Lucene query defined by Lucene query grammar.

Alternatively, any suitable search engine model may be employed tocreate the query-based classification rules without departing from thescope of the present disclosure. For example, the method 200 may employa Sphinx search engine, a DataparkSearch search engine, a Terrier searchengine, an OpenSearchServer search engine, etc.

As shown in block 222, query-based classification rules may be createdusing one or more user defined ad hoc categories from block 220. Forexample, for each user defined ad hoc category, a query-basedclassification rule may be created. The user defined ad hoc categoriesmay be created for a particular user so that the categories are relevantto that particular user. For example, a user defined ad hoc category maybe about a specific product or brand, an event regarding a product(e.g., a launch, a campaign, recall, etc.), etc.

The query-based classification rules are also created using thetopic-term matrix generated in block 210. In this way, relevant topicsmay be identified from the topic-term matrix for the query-basedclassification rule. Additionally, using the topic-term matrix assistsin identifying terms used by authors when writing about the relevanttopics.

In the example of FIG. 2, one or more test documents may be generatedbased on the processed training documents (block 224). In someembodiments, the document-topic matrix (block 208) and the topic-termmatrix (block 210) may also be utilized to generate the test documents.For example, the document-topic matrix and the topic-term matrix mayidentify general topics of each training document. This information maybe used to generate more relevant, targeted, etc. test documents fromthe training documents.

As shown in block 226, the Lucene Engine receives the query-basedclassification rules created in block 222 and the test documentsgenerated in block 224. The Lucene Engine indexes and matches the testdocuments to the query-based classification rules. The Lucene Enginethen outputs results (e.g., a relevance score) reflecting the relevanceof the test documents to the user defined ad hoc categories using thequery-based classification rules.

The results from the Lucene Engine may be evaluated in block 228 toensure each query-based classification rule is suitable (e.g., meets adefined threshold). For example, the generated test documents may beanalyzed by an individual and marked as relevant or irrelevant for eachuser defined ad hoc category (from block 220). Then, the query-basedclassification rule may be evaluated by utilizing precision and/orrecall equations as explained above. If the results are unsuitable, theclassification rule may be modified and evaluated again as explainedabove.

The query-based classification rules determined to be suitable may beidentified and stored for future use as explained below. For example,the query-based classification rules may be formatted and saved in XMLfiles. One example screen shot 700 of three query-based classificationrules in XML format is shown in FIG. 7.

In the example of FIG. 7, the three query-based classification rules areshown as Boolean Lucene Queries (“LuceneQuery”). The screen shot 700also includes user defined ad hoc categories (e.g., from block 220 asexplained above) and relevant topical categories from the topicalcategories generated by the topic model engine (block 206). For example,the topical categories may be analyzed (e.g., manually by a researchanalyst) in view of the query-based classification rules and thenrelevant topical categories may be identified and saved as an element ofone or more user defined ad hoc categories.

In the example of FIG. 7, the user defined ad hoc categories include“New England Auto Show,” Carbon Emissions” and “Car Recall.” Therelevant topical categories for “Carbon Emission” are “Auto” (Topic 12of FIG. 4) and “Green Energy” (corresponding to Topic 4 of FIG. 4 asexplained above). Therefore, only documents classified to topicalcategories “Auto” and “Green Energy” (as explained above with referenceto FIG. 6) may be matched to the Boolean Lucene query (i.e., thequery-based classification rule) for “Carbon Emissions.”

FIG. 3 illustrates a flowchart of an example method 300 for classifyingat least one electronic document using the topic model-basedclassification rules and/or the query-based classification rules createdin the method 200 of FIG. 2.

As shown in FIG. 3, the method 300 includes receiving electronicdocuments to be classified (block 302). For example, the electronicdocuments may be received as streaming data. The electronic documentsare then processed in block 304. For example, each electronic documentmay be filtered to clean textual content as explained above.Alternatively, the electronic documents may not be processed and insteadthe method 300 may classify unprocessed electronic documents.

In some embodiments, the electronic documents may be processed (block304) in substantially the same manner as the training documents wereprocessed in block 204 of FIG. 2. By processing the electronic documentsin substantially the same manner, the accuracy of classifying theelectronic documents may increase.

The processed electronic documents along with the topic-term matrix(block 210) and the topic model-based classification rules (block 218)created in method 200 may be provided to a topic predictor (shown inblock 306). In some embodiments, the topic predictor (block 306) may besimilar to the topic model engine (bock 206) of FIG. 2. However, thetopic predictor may not generate statistical data (e.g., the topic-termmatrix and/or the document-topic matrix) as may be done by the topicmodel engine of FIG. 2. Instead, the topic predictor may receive thetopic-term matrix (block 210) as explained above.

In the example of FIG. 3, the topic predictor may receive the processedelectronic documents from one or more other devices (e.g., servers,etc.) via a communication network (e.g., internet, intranet, etc.). Inthis way, the topic predictor may operate in real-time. Alternatively,the topic predictor may receive the processed electronic documents inany suitable manner including, for example, manually, via memory storingthe processed electronic documents, etc.

The topic predictor may classify the processed electronic documents byassigning one or more topical categories to the electronic documentsbased on the topic model-based classification rules. In this way, morethan one topical category may be assigned to a particular electronicdocument.

Alternatively, any suitable topic model engine may be employed toclassify the processed electronic documents without departing from thescope of the present disclosure. For example, the method 300 may employa probabilistic latent semantic indexing (PLSI) model, a latentDirichlet allocation (LDA) model, etc.

The topic predictor may assign the topical categories by establishing adistribution of topical categories of a particular electronic documentbased on the topic-term matrix (block 210). Then, based on theestablished topical distribution, the topic predictor assigns at leastone of the topical categories of the topic-term matrix to the particularelectronic document based on the topic model-based classification rules(block 218). In this way, the method 300 classifies the electronicdocuments via the topic predictor.

As shown in block 312, the method 300 then outputs the assigned topicalcategories for the particular electronic document.

In the example of FIG. 3, the method 300 employs the Lucene Engine(shown in block 226) as used in the method 200. The Lucene Engine may beemployed to classify the processed electronic documents based on thequery-based classification rules stored during the method 200 explainedabove. Alternatively, any suitable search engine model may be employedwithout departing from the scope of the present disclosure. For example,the method 300 may employ a Sphinx search engine, a DataparkSearchsearch engine, a Terrier search engine, an OpenSearchServer searchengine, etc.

In the example of FIG. 3, the Lucene Engine receives one or morerelevant query-based classification rules. As shown in block 308, therelevant query-based classification rules are selected from thequery-based classification rules (block 230) based on the assignedtopical categories from the topic predictor (block 312). For instance,if an assigned topical category of an electronic document is “fashion”or “sport,” then it may be unnecessary to check the relevance of theelectronic document with query-based classification rules regarding “carrecall.” However, if an assigned topical category of the electronicdocument is “auto,” then query-based classification rules related to“car recall” should be selected. In the example of FIG. 3, the relevantquery-based classification rules (block 308) are selected automaticallybased on analysis completed when the query-based classification rulesare created (e.g., during method 200). Alternatively, the relevantquery-based classification rules may be selected in any suitable manner.

For example, the method 300 may identify the relevant query-basedclassification rules by comparing the assigned topical categories fromthe topic predictor (block 312) with the relevant topical categoriessaved as elements of the user defined ad hoc categories as explainedabove with reference to FIG. 7. If an assigned topical category is thesame as a relevant topical category, the method 300 selects thequery-based classification rules (e.g., labeled “LuceneQuery” in FIG. 7)created for the user defined ad hoc categories from which the relevanttopical category is saved under. For example, if one of the assignedtopical categories from the topic predictor is “Auto”, the method 300selects the query-based classification rules (shown in FIG. 7) createdfor the user defined ad hoc categories “New England Auto Show,” CarbonEmissions” and “Car Recall” of FIG. 7. By selecting only the relevantquery-based classification rules, efficiency of the classificationprocess may improve, accuracy of the classifying electronic documentsmay improve (e.g., by reducing query ambiguity caused by homonym words,etc.), etc.

The Lucene Engine then may categorize the particular electronic documentfrom block 304 into at least one specific user defined ad hoc category(from the user defined ad hoc categories) using the selected relevantquery-based classification rules (from block 308). Therefore, theparticular electronic document may be classified by both assigningtopical categories to the particular electronic document and bycategorizing the particular electronic document in specific user definedad hoc categories in parallel.

The method 300 then outputs the specific user defined ad hoc categoriesfor the processed electronic documents. This is shown in block 310.

For example, if the selected relevant query-based classification rulescorrespond to the user defined ad hoc categories “New England AutoShow,” Carbon Emissions” and “Car Recall,” the Lucene Engine maycategorize the particular electronic document into any of the three userdefined ad hoc categories if the selected relevant query-basedclassification rules are satisfied. The Lucene Engine then outputs theuser defined ad hoc categories used to categorize the particularelectronic document. These user defined ad hoc categories are known asthe specific user defined ad hoc categories.

Alternatively, if the particular electronic document does not satisfyany of the selected relevant query-based classification rules, theparticular electronic document may not be categorized. As a result, theparticular electronic document may be classified only by assigningtopical categories to the particular electronic document (as explainedabove). Thus, the particular electronic document is classified onlybased on the topic model-based classification rules.

The assigned topical categories (block 312) from the topic predictor andthe specific user defined ad hoc categories, if any, (block 310) fromthe Lucene Engine may be employed by a user to identify topics anddocuments relevant to that particular user. In turn, the user mayutilize this information to more accurately collect data pertaining totheir business, interests, etc. As a result, the user is able to monitorrelevant discussions (e.g., positive and/or negative discussionsregarding a product, false accusations, etc.), monitor new topics and/orissues, respond to disgruntled customers, alter protocols, etc.

Additionally, the created topic model-based classification rules and/orthe query-based classification rules disclosed herein may be modified atany time during the process, including during or after the rule creatingprocess (e.g., method 200 of FIG. 2), during or after the electronicdocument classification process (e.g., method 300 of FIG. 3), etc. Forexample, the topic model-based classification rules and/or thequery-based classification rules may be modified after a period of time(e.g., after a period of minutes, hours, days, months, etc.). This inturn allows the topic model-based classification rules and/or thequery-based classification rules to more accurately reflect currenttopics discussed in electronic media sources.

FIG. 8 illustrates an example computer system 800 for classifying anelectronic document from an electronic media source. The computer systemincludes a communication network 802 and a computer server 804 incommunication with the communication network 802. The computer server804 receives electronic documents from electronic media sources via thecommunication network 802. In the example of FIG. 8, the computer server804 includes memory 806 and a processor 808.

The memory 806 may include training documents and user defined ad hoccategories as described above. The memory 806 may include onboard and/orexternal memory, including removable and/or remote memory. For example,the memory 806 may include cloud storage, disks, DVD, CD-ROMs, ROMs,RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, etc.

In the example of FIG. 8, the processor 808 generates statistical datafrom the training documents and creates at least one topic model-basedclassification rule using the statistical data and at least onequery-based classification rule using the user defined ad hoc categoriesand the statistical data. Then, the processor 808 classifies theelectronic document using the topic model-based classification rule. Insome examples, the processor 808 may classify the electronic documentusing both the query-based classification rule and the topic model-basedclassification rule as explained above.

Alternatively, the processor 808 may only classify the electronicdocument using a topic model-based classification and, if applicable, aquery-based classification rule created external to the system 800. Forexample, the classifications rules may be created external to the system800 (including the processor 808) and saved in XML files for future useas explained above. If desired, the XML files may be stored in thememory 806 and the processor 808 may then classify the electronicdocument using the topic model-based classification and, if applicable,the query-based classification rule saved in the XML files.

Additionally, although not specifically described, the processor 808 mayexecute any of the example methods described herein. For example, theprocessor 808 may evaluate the classification rules on test documentsstored in the memory 806 and then modify the classification rules ifneeded as explained above.

Further, the example methods disclosed herein may be implemented ascomputer-executable instructions. These computer-executable instructionsmay be stored on a non-transitory computer readable medium. In this way,the methods may be implemented in a computer program product stored on acomputer readable medium. The computer readable medium may include forexample, any type of disk including floppy disks, optical discs, DVD,CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs,EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems, including molecular memory ICs, or any type of system ordevice suitable for storing instructions and/or data.

For example, the non-transitory computer readable medium may includeinstructions for generating statistical data from one or more trainingdocuments, creating at least one topic model-based classification ruleusing the statistical data, and creating at least one query-basedclassification rule using one or more user defined categories and thestatistical data. The instructions stored on the non-transitory computerreadable medium may then be employed in a system (e.g., the system 800)to classify an electronic document from an electronic media source.

The documents disclosed herein may include any suitable electronicdocument representing, for example, social media articles (e.g., a blog,a message, etc.), news articles (print or electronic), emails, etc.Additionally, the electronic media sources disclosed herein may includeany suitable electronic media source including, for example, socialmedia, news outlets, etc.

The foregoing description of the embodiments has been provided forpurposes of illustration and description. It is not intended to beexhaustive or to limit the disclosure. Individual elements or featuresof a particular embodiment are generally not limited to that particularembodiment, but, where applicable, are interchangeable and can be usedin a selected embodiment, even if not specifically shown or described.The same may also be varied in many ways. Such variations are not to beregarded as a departure from the disclosure, and all such modificationsare intended to be included within the scope of the disclosure.

1. A method of classifying an electronic document from an electronicmedia source, the method comprising: generating statistical data fromone or more training documents, creating a plurality of classificationrules, including creating at least one topic model-based classificationrule using the statistical data, and creating at least one query-basedclassification rule using one or more user defined categories and thestatistical data, and classifying the electronic document using the atleast one topic model-based classification rule.
 2. The method of claim1 wherein classifying includes classifying the electronic document usingthe at least one topic model-based classification rule and the at leastone query-based classification rule.
 3. The method of claim 1 furthercomprising filtering each training document prior to creating theclassification rules.
 4. The method of claim 1 wherein creating includesevaluating a classification rule of the classification rules on a testdocument to ensure the classification rule meets a defined threshold. 5.The method of claim 4 wherein creating includes modifying theclassification rule if the classification rule is below the definedthreshold.
 6. The method of claim 1 wherein the statistical dataincludes a topic-term matrix and a document-topic matrix.
 7. The methodof claim 6 wherein the topic-term matrix includes one or more topicalcategories and wherein classifying includes assigning at least one ofthe one or more topical categories of the topic-term matrix to theelectronic document using the at least one topic model-basedclassification rule.
 8. The method of claim 7 further comprisingoutputting the assigned at least one topical category of the electronicdocument.
 9. The method of claim 1 wherein classifying includesclassifying the electronic document only using the at least one topicmodel-based classification rule.
 10. The method of claim 6 whereincreating includes creating the at least one query-based classificationrule using the topic-term matrix.
 11. The method of claim 10 furthercomprising selecting a query-based classification rule of the at leastone query-based classification rule based on the assigned at least onetopical category and wherein classifying includes categorizing theelectronic document into at least one user defined category of the oneor more user defined categories using the selected query-basedclassification rule.
 12. The method of claim 11 further comprisingoutputting the at least one user defined category of the one or moreuser defined categories.
 13. The method of claim 1 further comprisingmodifying the plurality of classification rules after a period of time.14. The method of claim 1 wherein the one or more training documentsinclude one or more unlabeled training documents.
 15. A non-transitorycomputer readable medium storing computer-executable instructions forcreating a plurality of classification rules to classify an electronicdocument from an electronic media source, the instructions includinggenerating statistical data from one or more training documents, andcreating a plurality of classification rules, including creating atleast one topic model-based classification rule using the statisticaldata and creating at least one query-based classification rule using oneor more user defined categories and the statistical data.
 16. A computersystem for classifying an electronic document from an electronic mediasource, the computer system comprising a communication network and acomputer server in communication with the communication network, thecomputer server configured to receive the electronic document via thecommunication network, the computer server having memory and aprocessor, said memory including one or more training documents and oneor more user defined categories, the processor configured to generatestatistical data from the one or more training documents, create aplurality of classification rules, the classification rules including atleast one topic model-based classification rule created using thestatistical data and at least one query-based classification rulecreated using the one or more user defined categories and thestatistical data, and classify the electronic document using the atleast one topic model-based classification rule.
 17. The computer systemof claim 16 wherein the one or more training documents include one ormore unlabeled training documents.
 18. The computer system of claim 16wherein said memory further includes a test document and wherein theprocessor is configured to evaluate a classification rule of theclassification rules on the test document to ensure the classificationrule meets a defined threshold.
 19. The computer system of claim 18wherein the processor is configured to modify the classification rule ifthe classification rule is below the defined threshold.
 20. The computersystem of claim 19 wherein the processor is configured to modify theplurality of classification rules after a period of time.
 21. Thecomputer system of claim 20 wherein the statistical data includes atopic-term matrix and a document-topic matrix
 22. The computer system ofclaim 21 wherein the topic-term matrix includes one or more topicalcategories and wherein the processor is configured to classify theelectronic document by assigning at least one of the one or more topicalcategories to the electronic document using the at least one topicmodel-based classification rule.
 23. The computer system of claim 22wherein the processor is configured to select a query-basedclassification rule of the at least one query-based classification ruleusing the assigned at least one topical category and wherein theprocessor is configured to classify the electronic document bycategorizing the electronic document into at least one user definedcategory of the one or more user defined categories using the selectedquery-based classification rule.
 24. (canceled)